OpenPack Dataset

@PerCom 2023 WS BiRD

We are hosting an activity recognition competition, OpenPack Challenge 2022, using the OpenPack dataset at a PerCom 2023 Workshop! The task is very simple: Recognize 10 work operations from the OpenPack dataset.

High recognition accuracy is required to introduce action recognition technology into the industrial domain. However, in packaging operations, not only the combination of items to be packed change a worker's movements greatly but also a number of other difficulties affect the packing process, such as a wide variety of motion speed, body movements, and procedures which are highly dependent on the skill level of the worker. Develop the perfect activity recognition model using OpenPack's various modalities and contribute to advancing smart logistics centers!

Start

2022-10-15 (AOE)

End

2023-01-15 (AOE)

Task

Recognize 10 Work Operations

(Temporal Semantic Segmentation)
Prize
  • Cash Prize
  • Travel Fee Support to attend Percom2023

Contents:

Overview

Background

Human Activity Recognition is beginning to be applied in industrial domains, targeting the improvement of manual labor efficiency. We know processes such as assembly and packaging inside factories or logistic centers still mainly depend on manual workers. To ensure a rapid response to changing demands, this trend is expected to continue. Therefore, quantifying manual labor is a crucial requirement to streamlining existing processes.

Workers inside logistic centers perform a series of sequential tasks to pack items. These working periods consist of actions like reading item labels or assembling the packaging boxes. Understanding details from each of these operations, i.e. its temporal duration or potential abnormalities, is crucial for optimizing the packaging process. However, period uniqueness related to item size or number of items included, make identifying such operation details a challenging task.

Challenge

You’ll develop a model to recognize the operations that conform packaging work with data from 4 IMU streams, keypoint data, etc. The packaging work consists of 10 operations (i.e., activity classes) described below. To quantify the operations as precisely as possible, dense labeling is required. You must predict activity classes for each 1second-long time slot. You can use data from 11 subjects to develop your model. The test data consists of 6 subjects (4 distinct subjects).

If successful, your work will help the ubiquitous research community improve current smart factories and better integrate human factors into the smart factory optimization process.

Plese visit the following page for more technical details such as task definition, evaluation procedures, and rules.

Prize

A. Cash Prize

The awards ceremony will be held at Percom 2023 WS (BiRD2023). We would like to hand out the cash prizes in this ceremony.

  • 1st: 100,000 JPY
  • 2nd: 40,000 JPY
  • 3rd: 20,000 JPY

B. Travel Fee Support

At least one person from the top-3 top-5 team strongly encouraged to attend the award ceremony. We will support the travel fee for the attendance up to 1 member (up to 750,000JPY/person) from each winner team.

Update: We were able to secure a budget for three people, so we changed our travel fee support from the top-2 to the top-3 teams. (2022-10-11)

Update: The top-5 teams will receive travel support. This means that in addition to the top three teams, one member from the 4th and 5th place teams will also be able to attend Percom2023 with travel support. In addition, the maximum support amount has been changed from 500,000 JPY to 750,000 JPY. (2022-12-31)

Winners Obligations

As a condition to being awarded a Prize, a top-5 winner must fulfill the following obligations. The detail instructions will be sent to top-5 winner after the final submission deadline.

  • Submit your code so that we can check for cheating.
  • Submit a short report paper that describes the award methodology.

Rules for OpenPack Challenge 2022

A. Registration Required

Please submit team registration form before you join this competition even if you a solo challenger. This is required to get prizes. There are no limitation for the team size. However, if you are creating multiple teams within the same lab, be careful not to violate the rules of PRIVATE SHARING.

When you want to merge your team with other team, please contact OpenPack Challenge Admin Team via email.

B. No Private Sharing Outside Teams

Privately sharing code or data outside of teams is not permitted. It’s OK to share code if made available to all participants on the forum.

C. External Dataset

You may use data other than the competition data to develop and test your submission. However, you will ensure the external data is publicly available to everyone without any cost.

Results

Thank you to everyone who participated, we had 33 teams and 106 registered userson colab. We received a total of 817 submissions during the period. (As of January 19, 2023)

The top five teams in the competition are listed below. Congratulations! Please see this table (Google Sheet) here for detailed scores.

1st Place

tomoon (F1=0.9633)

Tomoki Uchiyama. (Tsukuba Univ., JA)

2nd Place

vbu211 (F1=0.9592)

Yuto Namba(a,b), Yuichi Nakatani(a), Kenta Ishihara(a), Sachio Iwasaki(a), Kosuke Moriwaki(a), Xian-Hua Han(b), Tetsuo Inoshita(a). (a=NEC Corporation, JP; b=Yamaguchi Univ., JP)

3rd Place

Ritsumei (F1=0.9241)

Shurong Chai(a), Jiaqing Liu(a), Rahul Kumar Jain(a), Yinhao Li(a), Tomoko Tateyama(b), Yen-Wei Chen. (a=Ritsumeikan Univ, JP; b=Fujita Health Univ., JP)

4th Place

Malton (F1=0.9171)

Yusuke Matsubayashi. (Osaka Univ., JP)

5th Place

Shubham Wagh (F1=0.9112)

Shubham Maroti Wagh. (Veridium in Oxford, UK)

All Results

Rank Team F1-measure
1 tomoon 0.9633
2 vbu211 0.9592
3 Ritsumei 0.9241
4 Malton 0.9171
5 Shubham Wagh 0.9112
6 UCLab 0.9057
7 liuqijd 0.8987
8 Potros 0.8822
9 Dialga 0.8752
10 SotaroFushimi 0.8466
11 Tetsu_roo 0.8118
12 syoka4156 0.8114

Awards Ceremony @Percom2023 WS BiRD

The awards ceremony was held on March 13, 2023 at Percom2023 WS BiRD. The top 5 winners gathered to discuss their solutions in a poster session. Congratulations again!

Picture with Top-5 Winners @Percom2023 WS BiRD

Registration and Submission (Closed)

Before submitting your prediction, registration is required. You need to register your information to both (1) CodaLab and (2) Google Form. Please follow the procedures below.

  • Solo challengers are also welcom! But don't forget to register you as a solo team!
  • No limitation to team size. (But be careful about violation of NO PRIVATE SHARING policy when you make multiple teams.)

Steps for Registration

  1. [Every Members] Make your codalab accounts. All members should have thier own accounts.
  2. [Every Members] Register to this challenge from Participate tab in the codalab page.
  3. [Team Leader] Submit the Google Form for team registration.
  4. (Wait for a while... Admin team will check and aprove your team.)
  5. You got email from codalab about approval and you can submit your prediction!

Submission & Leadersboard

Timeline

2022-10-01
Registration Open
Google form and competition page on codalab will be public.
2022-10-01
Registration Open
Google form and competition page on codalab will be public.
2022-10-15
Launch
You can submit your estimates to the competition site.
2022-10-15
Launch
You can submit your estimates to the competition site.
2023-01-10
Entry and Team Merger Deadline
If you want to join this competition or change team members, you must register until this date.
2023-01-10
Entry and Team Merger Deadline
If you want to join this competition or change team members, you must register until this date.
2023-01-15
Final Submission Deadline
You must submit your best results until the end of this day.
2023-01-15
Final Submission Deadline
You must submit your best results until the end of this day.
2023-01-31
Report Submission Deadline
Top 3 winners must submit a report of your solution to get the prize! Deadlines are subject to change.
2023-01-31
Report Submission Deadline
Top 3 winners must submit a report of your solution to get the prize! Deadlines are subject to change.
2023-03-13
Workshop Day @Percom2023
Award Ceremony will be held. Top 3 winners are requested to participate offline.
2023-03-13
Workshop Day @Percom2023
Award Ceremony will be held. Top 3 winners are requested to participate offline.

Get Updates from OpenPack Team!

Get the latest information about the OpenPack Challenge or OpenPack Dataset on our mailing list (Google Group), Twitter and GitHub. Please subscribe and follow now!

Getting Started

Notebooks (Colab / Jupyter Notebook)

Video

Tutorial Session (日本語)
Beginner
This is a tutorial session for the OpenPack Challenge 2022, held on October 28, 2022, at the academic workshop (Hierarchical BioNavigation Workshop). The session explains how to build a work activity recognition model and how to submit it to the competition site (codalab), which is the challenge of the competition.

Slide

GitHub

OpenPack Toolkit

dataset utilities and documents

GitHub

OpenPack Torch

sample code (PyTorch)

YouTube

OpenPack Dataset Channel

sample videos

Codalab: System for Competition

OpenPack Challenge 2022

Submission and evaluation system for this challenge.

Members

Takuya Maekawa
Osaka Univ.

Naoya Yoshimura
Osaka Univ.

Jaime Morales
Osaka Univ.

Adviser

S

Sozo Inoue
Kyushu Institute of Technology

K

Kazuya Murao
Ritsumeikan University

Sponsors

Sponsor.1 (TBA)
Cynav: What is Hierarchical Bio-Navigation?