Property Finder Fixed AI Evaluation Bias with Amazon Bedrock
Property Finder transitioned to the Llama 4 Maverick 17B Instruct model on Amazon Bedrock after detecting evaluation bias in its AI models. In the process, the company reduced its costs by 86 percent and achieved higher accuracy rates.
Property Finder determined that 96.8 percent of the evaluation labels in its AI system, which processes tens of thousands of real estate images daily across the Middle East and North Africa, were inherited from a single model's predictions, leading to bias.
To solve this issue encountered while migrating its vision AI workload to Amazon Bedrock, the company created a human-verified baseline dataset. By manually labeling selected images from the United Arab Emirates and Egypt, the bias loop based on the existing model's predictions was broken.
Collaborating with the AWS Generative AI Innovation Center, the team developed the final prompt named 'v27_PF' through an iterative optimization process. As a result of this work, the Llama 4 Maverick 17B Instruct model achieved a Micro F1 score of 0.80 on a test set consisting of 356 images, surpassing the previous model's score of 0.56.
Sources
- aws.amazon.com · Link