Michael_Lippe - Hardware Engineering (Level 1) Pathway

Week 1 Self Assessment

Technical Area

  • Learned about the concept of FPGAs
  • Learned about the specifics of the PYNQ Z2 board
  • Learned about premade IP and how to make your own IP for the PYNQ board using Vivado.

Tools

  • Learned how to interface Jupyter Notebook with the PYNQ board and refreshed my knowledge of it
  • Learned how to use the PYNQ Terminal to download example notebooks from Github
  • Learned how to access the PYNQ Serial Terminal from the macOS Terminal

Soft Skills

  • Introduced myself to the Session 1 Leads and helped organize troubleshooting of the PYNQ boards
  • Brainstormed remote collaboration ideas with Stem-Away staff and the other leads
  • Communicated the next steps for hardware to interns on both Discord and the StemAway Form after discussion with other leads

Achievements

  • Got the PYNQ board fully up and running
  • Accessed the board’s Serial Terminal and used it to modify the network settings of the board
  • Used Github and Jupyter Notebook to install and test example code on the board

Goals for the Upcoming Week

  • Help everyone get their PYNQ boards up and running
  • Come up with tentative ideas for the main project
  • Help organize remote collaboration and facilitate the pairing process
  • Experiment with running my own Python code on the board

Tasks in Detail

  • Got PYNQ Board up and running: Had trouble flashing MicroSD card at first but was fixed by updating my copy of balenaEtcher. Also had trouble resolving the host name of the board so used the Serial Terminal to modify the network settings of the board and find the board’s given IP address. Since the board presents two different serial ports to the computer, I had to figure out which one was for the terminal or else I just got garbage on the screen. Turns out it was the second port.
  • Installed example code on the board from Github: Had issues with the default file location being root directory rather than the Jupyter Notebook folder so used the Terminal to navigate to the correct folder, also had to update the Python and GitHub packages.
  • Added PYNQ board as a network drive to my computer: Had to modify the default server address to use the Board’s new IP address as I had disabled the board’s static IP address so it would better interface with my network.
  • Communicated the information from the Team Leads meeting to the rest of the interns and helped design a Google Form to help pair up students with and without boards: Since not everyone is in the hardware team Discord yet we decided it was best to post the form on both the Discord and StemAway form while also reminding people to join the discord.
1 Like

Week 2 Self Assessment

Technical Area

  • Learned about the OpenCV library
  • Learned about IPython Widgets
  • Learned how to read the switches on the PYNQ board
  • Learned how to connect a USB webcam to the PYNQ board

Tools

  • Learned how to setup and invite people to Trello boards
  • Learned how to access the PYNQ Serial Terminal using Windows rather than macOS in order to help people having trouble with it

Soft Skills

  • Organized and planned weekly team meetings
  • Came up with an idea for one of the two project teams
  • Answered questions and helped initiate a new member
  • Took initiative by creating the team choice form, attendance spreadsheet, and Team Trello boards
  • Helped others run my webcam notebook

Achievements

  • Modified an existing Jupyter Notebook to display live video from an attached webcam
  • Came up with the main project idea for Team 1 of a real life Pokedex
  • Helped run team meeting

Goals for the Upcoming Week

  • Help everyone get familiar with creating Jupyter notebooks and using OpenCV
  • Post team rosters and board pairings
  • Create road maps for the two main projects
  • Sort out the lack of activity from hardware participants so we can start moving ahead with the main projects

Tasks in Detail

  • Modified a Notebook to display live video from a webcam: Initially I wanted to use an HDMI monitor only to display the video but since I did not have access to one at the time I had to get it to also display in the Jupyter Notebook. I initially tried to use matplotlib like the example notebook did to display images but quickly realized it wouldn’t work for video, so I googled for a solution and came upon IPython Widgets and was able to use that to display the video. I also knew I wanted the switches to control the program and video feed so I learned how to read them from another example notebook.
  • Organized and ran the first weekly team meeting: I helped announce the meeting, draft and outline for it, and create slides for it. In the parts of the meeting where we were waiting for people to fill out forms it was a bit awkward with all the silence so I came up with the idea of putting on background music to help ease the tension.
1 Like

Week 3 Self Assessment

Technical Area

  • Learned how to import functions from one Notebook into another
  • Learned the required format of images for a CNV training dataset
  • Learned about the differences between BNN’s CNV and LFC networks
  • Learned about panda and using it to more easily import CSV files into Python programs

Tools

  • Learned how to record operations in Photoshop and automatically apply them to folders of images
  • Refreshed my knowledge on creating and sharing a Github repository
  • Used Quora, Reddit, and other sites to find image databases and database repositories

Soft Skills

  • Organized people on Team 1 into two groups, one of which will work on the Machine Learning part of the project and the other the Data collection and display part
  • Helped troubleshoot various PYNQ board problems and teach people how to use the board’s serial terminal
  • Distributed tasks to team members, mainly working on the animal information database, and explained future steps

Achievements

  • Created an outline program for the Pokedex project and got the image capture part fully functional
  • Started building an animal image database for use in training our image classifier model
  • Helped run the second weekly team meeting and fully ran the Team 1 portion of it

Goals for the Upcoming Week

  • Finish animal information database and make significant progress on accessing the information in Python
  • Finish the animal image database and start training the image recognition model
  • Start work on displaying text over the PYNQ board’s HDMI out

Tasks in Detail

  • Built an outline program for Pokedex project: Had to figure out how to import functions from other notebook files since they are more complex than standard .py files and thus need a special library to be imported properly. After then I had issues because I was importing the base overly in all of my notebooks which caused them to error so I passed through the base variable from the Main notebook to the other ones and that solved the issue. Finally, BNN kept crashing the kernel and I couldn’t figure out why. I spent hours trying different code and looking at the example notebooks, even uninstalling and reinstalling BNN but nothing fixed it. I’m not gonna worry about it too much since we will use our own custom classifier anyway which hopefully won’t face the same issues.
  • Started building an animal image database: I collected presorted images of animals from various public databases and compiled them into a single database with 90 animal classes. After manually sorting through the photos to remove any unrelated photos that ended up in the class’ folder, I ran all the images through Photoshop to resize them to 32x32, the required format for CNV.

Week 4 Self Assessment

Technical Area

  • Learned how to train my own image classifier using BNN
  • Learned how to accelerate the training using Nvidia Cuda and cuDNN
  • Learned how to set up the BNN training environment
  • Learned how to resize images using Python and the PIL library
  • Learned about the Pandas library and how to use it to read CSV files
  • Learned how to isolate Python 2 from Python 3 and install libraries to each specifically

Tools

  • Learned about Kaggle as a way to find free to use image datasets
  • Learned how to use a Linux environment on Windows
  • Learned how to dual boot Linux and Windows
  • Learned how to assign specific people to Trello tasks and assign those tasks due dates

Soft Skills

  • Worked with other team leads to assign people tasks to fill out entries in the animal information database
  • Helped other people convert images to the proper format for use in training a neural network with BNN
  • Added specific future tasks to Trello so people could see what was coming up in the near future

Achievements

  • Installed the BNN training environment on Linux
  • Started training a neural network to classify images of various animals
  • Tweaked various parameters of the training program to optimize the neural network
  • Helped build an animal information database

Goals for the Upcoming Week

  • Finish training the neural network and get it to below 10% error
  • Help team build a program to extract information from the animal database as a CSV file
  • Figure out the specific details of how to continue the project after Session 1 leaves
  • Work on displaying text alongside images over HDMI using the PYNQ board

Tasks in Detail

  • Installed the BNN training environment: After initially trying to install the required programs on Windows through an Ubuntu environment, I decided it was easier to just install Ubuntu and do it there. Installing Cuda and cuDNN required hours of troubleshooting across multiple days as various packages were either misbehaving or failing to install. Eventually I got everything installed but the BNN training program was incompatible with Python 3 which is where all my packages were going. Trying to uninstall Python 3 broke the Ubuntu install. I then reinstalled Ubuntu, installed all the software again, and figured out how to make pip default to python 2 and thus install packages there.
  • Started training a neural network to classify images of various animals: After changing the provided code so that it used my dataset instead of the default cifar10 I ran into issues where I had to first convert my dataset to base32 and keep it under 63 classes. Once the training was actually started the accuracy kept being much lower than desired so I decided to remove a ton of classes from the dataset and try again. This improved the accuracy but not enough so I started building a larger dataset with many more images per class. I am currently finishing the dataset and hopefully, it will provide much better results.
  • Helped build animal information database: Using Google Sheets each animal in the image dataset was given a row and various info about them was entered. Some initial categories such as Genus ended up being too complicated so it was replaced with Conservation Status. Everyone on the team was required to fill out entires so we could complete it.

Week 5 Self Assessment

Technical Area

  • Learned how to use PIL to overlay text and images onto other images
  • Learned how to use gTTS to convert text to speech
  • Learned how to use pydub to convert MP3 files to WAV files
  • Learned how to generate binary weight files for a trained neural network and upload those files to the PYNQ Board

Tools

  • Learned how to use CLONEZILLA to clone Linux partitions
  • Learned how to install Vivado
  • Learned how to resize and shave images using Python

Soft Skills

  • Reached out to people to confirm if they were staying for Session 2
  • Created and posted instructions for displaying the information from the animal database as text on a stylized background
  • Helped troubleshoot PYNQ board problems and programming problems

Achievements

  • Created a larger dataset with 2000 images per class to improve the model accuracy
  • Finished Training the Neural network using the new larger dataset
  • Generated Binary Weight files and uploaded them to the PYNQ Board
  • Started rebuilding the BNN hardware overlay

Goals for the Upcoming Week

  • Finish rebuilding the BNN hardware overlay and upload it to the PYNQ board
  • Fix errors with text to speech program
  • Help finish the information display

Tasks in Detail

  • Created Larger Dataset of 2000 Images Per Class: After getting images from dozens of public databases some classes still needed more images. To overcome this I used a tool that allowed me to download images from google images automatically. This allowed me to find images much quicker than before.
  • Finished Training Neural Network: After converting the new image set, the Neural Network took 3 days to train after which we got an error rate of around 41%. This was deemed good enough for the project as any further improvement would require significantly longer training sessions.
  • Generated Binary Weight files and uploaded them to the PYNQ Board: After figuring out where exactly jupyter stores files on the PYNQ Board I was able to upload the Binary Weight Files and use the serial terminal to move them to the correct location. However, when I ran the program it crashed and I soon learned that I had to rebuild the hardware overlay because we added more than 10 classes.
  • Started Rebuilding Hardware Overlay: I had to download Vivado for this but because it took up so much space it needed to be on a separate drive. This ended up causing issued when running the program to rebuild the hardware overlay so I am currently cloning the Linux install to another drive with more space and will reinstall Vivado there

Week 6 Self Assessment

Technical Area

  • Learned how to rebuild the BNN hardware overlay and upload it to the PYNQ Board
  • Learned how to use the PYNQ Board’s Line-In to record and save audio
  • Learned how to display an image using the PYNQ Board’s HDMI Out
  • Learned how to modify the outgoing HDMI frames from the PYNQ Board

Tools

  • Learned how to install older versions of Vivado
  • Learned how to set temporary variables in the linux terminal
  • Learned how to use the linux terminal to copy folders and modify directories

Soft Skills

  • Helped walk people through the weekly task of creating the animal information display
  • Communicated with mentor to set up round table discussion for Team 1
  • Commented and walked though current code with other interns

Achievements

  • Rebuilt the BNN hardware overlay and uploaded it to the PYNQ board
  • Got classifier functional within the pokedex program
  • Added border to camera capture screen
  • Recorded “animal not recognized” audio and made a function to play it
  • Combined all current programs into a single main program and finalized much of the user experience
  • Guided participants to create the animal information display

Goals for the Upcoming Week

  • Fix the remaining errors with the text to speech program
  • Help participants add a tracker for the number of times a species has been encountered
  • Finalize all code, text, and images as much as possible

Tasks in Detail

  • Rebuilt the BNN hardware overlay and uploaded it to the PYNQ board: This task ended up requiring Vivado HLS which isn’t included in the newer versions in favor of Vitis HLS. To fix this I downloaded an older 2018 build of Vivado which seemed to work. When this was done I uploaded the binary files to the PYNQ board but the classifier still crashed every time. This ended up being because the classes.txt file only contained the first 10. However, I couldn’t just put in the classes at random as I had to figure out what order they were in. To do that I just made a program to batch process images of each animal and return the most common class number for each set. This worked and I was able to add all 35 classes in the correct order.
  • Got classifier functional within the pokedex program: At first I had wanted to use the hardware version of the classifier but I quickly realized that was impossible. Since the program used the base overlay to access HDMI, buttons, switches, LEDs, and audio I couldn’t load the BNN hardware overlay since there was one already in use. Lacking the skills to combine the two overlays in Vivado I resorted to using the software version of the classifier. While it was slower it made little to no difference given the fact that only one image was being processed at a time.
  • Added border to camera capture screen: The HDMI output was 1080p but when the webcam was set to that resolution the frame-rate would drop significantly. To fix this the camera was set to 576p and the image was centered in the frame. To fill the empty black space I used a cool red background pattern I found and added some text telling the user to press BTN3 to take a photo. Initially I wanted a base frame that was modified with the background image and then to overlay new camera frames onto that as to not waste resources on modifying parts of the screen that don’t change. However, even after using deepcopy the frame would just end up being a bunch of garbage so I had to create a new frame every time. This ended up being fine and introduced no noticeable lag.
  • Recorded “animal not recognized” audio and made a function to play it: I wanted the board to play an error message when the class returned was other, as this meant no animal was recognized and the user would need to take another image. With the text to speech program still not working I decided to record the message using the Board’s Line-In. After making the mp3 file with GTTs I connected my computer’s line out to the PYNQ board’s line in and after much tinkering was able to record and save the audio. This then let me play it back without error.
  • Combined all current programs into a single main program: All the functions so far were modified and put into three separate notebook files which were being imported into the main notebook. After creating two functions in that main notebook to further organize the code it was all tested and confirmed working.

Week 7 Self Assessment

Technical Area

  • Learned how to run Jupyter Notebooks from the command line
  • Learned how to create and execute bash files
  • Learned how to use Systemd to create startup services

Tools

  • Used the PYNQ support forms to learn how to automatically start a program on boot

Soft Skills

  • Created a Google Slides template for team members to fill in

Achievements

  • Recorded Animal Info Audio for each Animal
  • Incorporated the species encounter tracker code into the main program
  • Finalized code, text, and images for main program
  • Started work on Final Presentation

Goals for the Upcoming Week

  • Record Video of Pokedex in action
  • Finish Final Presentation

Tasks in Detail

  • Attempted to create startup service for Pokedex Program: Ideally the Pokedex program would start at boot instead of having to be manually triggered. First, a bash file was created which contained the commands to run the notebook files from the command line. The bash file was tested and it worked. This was then incorporated into a startup service but no matter what I did I couldn’t get it to work. I tried disabling the startuppy service which is responsible for flashing the LEDs as it could override my code but that didn’t help. After trying countless suggestions I determined I had to move on and work on finalizing the code.
  • Recorded Animal Info Audio for each Animal: Since I couldn’t get text to speech to work live, I instead pre made mp3 files for each animal. These audio files were then played off my computer and recorded using the PYNQ board’s line in. They were then saved to a folder. I had to play around with the volume and length settings but eventually i got good recordings.
  • Incorporated the species encounter tracker code into the main program: The task for the participants for the last week had been to modify the function that displayed information about an animal to include a section about the number of times a species had been encountered. The code worked in isolation but I had to make a few modifications from what I was presented with. I changed some file names, variable names, and other small things that were causing errors when put into the main program.
  • Finalized code, text, and images for main program: With everything working, I went through all our code to remove any debugging lines, unused functions, and other pieces we didn’t need. I then tried to make all the variable and function names as consistent as possible and clean up the comments. The finalized code was then uploaded to our Github Repo.