Perspectives on Large Language Model (LLM) Evaluation

Details
When you’re choosing an LLM to deploy for your use case, there are multiple considerations. Does it need to be open-source, or can you use a proprietary model? Do you need to be able to fine-tune it? How much does it cost, and would you rather pay for usage by the token or be able to host it yourself? And, of course, how does it actually perform on the tasks that you’re planning on using it for?

For this panel, which is co-sponsored by Women Who Code DC and Data Science DC, we’ll bring together experts to discuss model selection, including choice of benchmarks and how to evaluate models for your specific use case. We’re still working on finalizing the panel speakers, but we’ll have at least one data scientist/machine learning engineer, as well as other people working on testing models.

This event will be in-person. We will try to add an online streaming option if the space and hardware will accommodate. Check back here at 6:15 p.m. ET on the evening of the event and if there's a link here, then that's an option. If there's not, then it's not. Same goes for recording.

Agenda
6:00 - Food and networking
6:30 - Panel Questions and Q&A
8:00 - Closing Remarks

Speakers (additional speakers TBA):

Prema Roman is a distributed systems engineer at Rotational Labs. She is an experienced software, data, and machine learning engineer with a proven track record of building high quality software applications and data products. Her passion for continuous learning has taken her a long way from her start as a data analyst, as she takes on new challenges at Rotational Labs building globally distributed systems and machine learning data products.

Pri Oberoi (she/they) is Staff Data Scientist at Axios HQ, where she shapes and develops ML features that help people write, send, analyze and align on internal business comms. Across their experience in non-profits, government, and industry their focus has been on building user-centric products. Pri works with Product, Design, the Executive team and an internal team of communication experts to shape the product roadmap and measure how ML features affect key product metrics.

Paula Gearon is a Semantic Architect at a medical data company. She has implemented database systems for much of her career, focusing on graph databases and functional programming. She has also been active in the Semantic Web and the W3C standards processes.

Bill Frischling is a Distinguished Scientist and Vice President of Emerging Technologies at FiscalNote. He started working on the Internet 17 days after Netscape released its first web browser, and has described his career as "yelling at computers." However, since they now yell back, he's looking for a different description. If he needed to fit into a certain shape, "applied engineering" fits best. He's been applying it with AI and ML tools for more than a decade.Frischling started his career as a reporter at The Philadelphia Inquirer, then helped launch The Washington Post on the Internet in 1996. He has run products and groups at The Post, AOL, Gannett and U.S. News and World Report. He's founded four startups, sold two, allows one continue to run itself, as it has since 1999, and never, ever discusses the fourth one. He's been working with AI for more than a decade, primarily because he likes when computers do his job for him.

Anastassia Kornilova (moderator) is the Director of Machine Learning and Founding Engineer at Trustible AI, a platform for Responsible AI Governance. In this role, she researches the benefits and risks of AI technologies and helps companies prepare for upcoming AI regulations. Previously, she has worked on building Machine Learning systems across multiple sectors with a focus on Legal NLP.

Our Code of Conduct
Women Who Code (WWCode) is dedicated to providing an empowering experience for everyone who participates in or supports our community, regardless of gender, gender identity and expression, sexual orientation, ability, physical appearance, body size, race, ethnicity, age, religion, socioeconomic status, caste, or creed. Our events are intended to inspire women to excel in technology careers, and anyone who is there for this purpose is welcome. Because we value the safety and security of our members and strive to have an inclusive community, we do not tolerate harassment of members or event participants in any form. Our Code of Conduct (https://www.womenwhocode.com/codeofconduct) applies to all events run by Women Who Code, Inc. If you would like to report an incident or contact our leadership team, please submit an incident report form (http://bit.ly/wwcode-incident-form).