Benchmark Program Manager -REMOTE

Job Locations US-Remote
Posted Date 1 week ago(5/19/2022 4:57 PM)
Program Operations Manager



Benchmark Program Manager


Position Summary


Voice assistants are becoming one of the most pervasive technology tools and today underserve many diverse populations. Appen’s Voice Assistant Benchmark Program aims to help Voice Assistants be more inclusive and less biased starting with African American English (AAE) by creating a highly-engaged and remunerated community of Americans who will work to create data that powers ASR systems. Using cutting-edge research on distinct AAE groups, smart speaker performance testing and at-scale capability in building crowds, Appen aims to create the first linguistically-representative crowd of speakers of AAE.

Benchmark Program Manager will join the team of AI Specialists to oversee Appen’s Voice Assistant Benchmark Program, design and implement benchmark-related quality assurance methodology, manage language data collection and annotation tasks for related projects, support with resource training, participants onboarding, management for the community of AAE speakers, as well as working with the Community Manager to ensure regional variation inclusivity through engaging with representative demographic varieties across distinct sub-regional linguistic groups.




  • Assist with designing, building and managing Appen’s AAE Voice Assistant Benchmark Program data collection and annotation workflows.

  • Monitor AAE language annotation and content transcription processes as part of the Voice Assistant Benchmark Program and other related initiatives.

  • Conduct research and provide linguistic consultancy on AAE.

  • Manage crowd participation strategy, production bandwidth, capabilities and resource utilization.

  • Outline and implement quality assurance methodology and validation checks for AAE transcription data.

  • Design and document training and onboarding plan for new participants of AAE-related projects.

  • Advise on effective resourcing strategies to support the AAE Voice Assistant Benchmark Program.

  • Manage AAE-related public communication pertaining to Appen’s Voice Assistant Benchmark Program.

  • Work with a team of linguists to develop letter-to-sound rules for Automatic Speech Recognition (ASR) in AAE.

  • Present demographics for speakers of AAE and program participation strategy in linguistic terms.

  • Apply inclusivity and fair sampling practices to the  community of speakers of AAE design and program participation management strategy to ensure representative assessment of Voice Assistants and other ASR systems.


Required Knowledge, Skills and Abilities


  • Minimum BA (major in Linguistics).

  • The candidate must be a speaker of African American English.


Professional Qualifications and Experience


  • 2 years of professional experience in program or project management.

  • Academic or industry experience in phonetics or phonology.

  • Relevant experience in language or data annotation services strongly preferred.


About Appen


We at Appen look for individuals who embody our core values: performance, honesty, humility, and grit. At Appen, we embrace and strive purposefully with our core values in mind throughout every aspect of our work and expect our employees to meet and exceed these standards. If this resonates with your work ethic and style, we encourage you to apply for this Appen position.


Appen collects and labels images, text, speech, audio, video, and other data used to build and continuously improve the world’s most innovative artificial intelligence systems. Our expertise includes having a global crowd of over one million skilled contractors who speak over 180 languages and dialects, in over 70,000 locations and 130 countries, and the industry’s most advanced AI-assisted data annotation platform. Our reliable training data gives leaders in technology, automotive, financial services, retail, healthcare, and governments the confidence to deploy world-class AI products. Founded in 1996, Appen has customers and offices globally.


At Appen, we value performance, honesty, humility, and grit. We persevere and remain focused, whilst maintaining agility to achieve quality outcomes and exceed expectations. We’re truth tellers – respectfully of course. We take accountability for our actions, and believe in giving and receiving direct feedback. We give credit where credit is due, and show gratitude to others for their contributions. We seek diverse perspectives as we recognize the value in teamwork and collaboration. Through grit, we take ownership, and we don’t give up.


We are an equal opportunity employer and value diversity at our company. We do not discriminate on the basis of race, religion, color, national origin, gender, sexual orientation, age, marital status, veteran status, or disability status.




Sorry the Share function is not working properly at this moment. Please refresh the page and try again later.
Share on your newsfeed