PalmX Logo PalmX 2025

PalmX 2025: The First Shared Task on Benchmarking LLMs on Arabic Culture

This shared task at ArabicNLP 2025 focuses on evaluating Arabic cultural understanding through two subtasks: General Culture and Islamic Culture. Using a high-quality, MCQ dataset in MSA and dialects, the task promotes the development of culturally aware models tailored to the diverse heritage of the Arab world.

Frequently Asked Questions

Common questions about participating in the ArabicNLP 2025 Shared Task.

Guidelines for Participating Teams

Who can participate in shared tasks?

Anyone who wants to can participate, either independently or by forming a team.

Is there a fee to participate in a task?

No, participation is completely free but at least one team member should pay to register for the conference.

How can I sign up to participate in a task?

Please see instructions on the website for the task (e.g., on CodaLab or other evaluation platforms).

Can I participate in multiple shared tasks?

Yes, you can participate in multiple different tasks.

Can I participate in multiple teams under the same shared task?

This depends on the task. Check with the task organizers to see whether they allow an individual to be part of multiple teams within the task.

What does "evaluation period" mean?

The evaluation period is the time window in which the official part of each task competition takes place. Task organizers will release evaluation data to participants at the beginning of the evaluation period, and system outputs will be due before the end of the evaluation period.

How many runs can be submitted for a task?

This depends on the task. Task organizers often specify how many submissions by each team will be evaluated. Contact your task organizers if these details are not already stated on the task webpage.

Do I need to write a paper?

Yes, if you register and request the dataset, you agree to submit a paper. It's not just encouraged—it's a binding commitment you make when receiving the data.

What is the paper format?

Shared task papers must follow the format for short papers at ArabicNLP 2025. Papers must follow official EMNLP (ACL) LaTeX or Word templates, be up to 4 pages (excluding unlimited references). Check the dedicated paper submission guidelines URL below for more details: https://github.com/acl-org/acl-style-files

Must someone from my team attend the conference?

Yes, at least one of the shared task team members should register and attend the conference.

Should I release my code?

You are highly encouraged to release the code or data used and include the URL in the paper. This promotes reproducibility.

My question isn't answered here. Who should I contact?

  • For task-specific questions: Consult the task website and contact task organizers.
  • For workshop logistics: Contact the shared task organizers.
  • For OpenReview technical issues: Contact OpenReview support.