Embedding Matrix Evolution in Vision Transformers

Last summer, I had the opportunity to work at the Graphics Imaging and Light Measurement Lab at Columbia University, under the mentorship of Professor Corey Toler-Franklin. Alongside a graduate student team, I contributed to the development of a novel vision transformer architecture designed for tumor detection in medical slides. My primary responsibility was to conduct an experiment aimed at understanding how transformers learn during the fine-tuning process when trained on our medical dataset. I achieved this by generating and analyzing comparisons of the embedding matrices before and after fine-tuning. My research provided valuable insights into the internal mechanisms of the models, which are often seen as black-box technologies, and greatly enhanced the team's understanding of the models we're improving. I am excited to continue working with the lab this fall.
Technical Skills
- Utilized the open-source MMDetection framework within a Docker container on AWS to train and test models
- Visualized and analyzed data using Pandas and Matplotlib
Non-Technical Skills
- Designed a research poster to present at the Columbia Undergraduate Research Symposium in October 2024
- Independently learned the open-source framework and trained other lab members on how to use it