OpenPack Dataset

What is OpenPack dataset?

OpenPacking Dataset is a new large-scale multi-modal dataset of packing processes. OpenPack is an open access logistics-dataset for human activity recognition, which contains 53 hours of human movement data with 20,129 instances of work operation annotation. In addition, OpenPack dataset contains data from IoT-enabled devices and rich metadata such as error annotation and order information.

The latest release is OpenPack Dataset v1.0.0. (Release Note)

Paper: N. Yoshimura, J. Morales, T. Maekawa and T. Hara, "OpenPack: A Large-Scale Dataset for Recognizing Packaging Works in IoT-Enabled Logistic Environments," 2024 IEEE International Conference on Pervasive Computing and Communications (PerCom), Biarritz, France, 2024 (link)

Characterisitcs

53H+

Large Scale activity dataset in Industrial Domain

OpenPack is the first large-scale dataset for packaging work recognition. 16 distinct subjects packed 3956 items in 2048 shipping boxes in total and the total recording length is more than 53 hours. The work operations and actions were annotated by the professional annotators. The total number of annotated work operations and actions are 20,129 and 52,529 respectively.

9
+ IoT

Rich Modalities - Vision + Wearable + IoT Device

OpenPack provides 9 data modalities including acceleration, gyroscope, quaternion, blood volume pulse (BVP), and electrodermal activity (EDA) data, as well as keypoints, a LiDAR point cloud, and depth images. As for the wearable sensor modality, we collected data with 4 IMUs and 2 E4 sensors. For the vision modality, 2 depth cameras (front view and top view) and 1 LiDAR sensor were installed to our environment. In addition, we collected data from IoT devices which are sparse but have strong relation with specific activity classes.

META

Rich Meta Data

OpenPack also provides a rich set of metadata such as subject’s experience in packaging work and physical traits of subjects, enabling designs of various research tasks such as assessment of worker’s performance in addition to basic work activity recognition. In addition, we made a list of irregular activities which can be used for developing normal detection technologies.

Sample Video

Acitivty Classes

Work Operations (10 classes)

ImageIDOperation
100Picking
200Relocate Item Label
300Assemble Box
400Insert Items
500Close Box
600Attach Box Label
700Scan Label
800Attach Shipping Label
900Put on Back Table
1000Fill out Order
8100Null

Data Collection Environments & Sensors

Download

OpenPack dataset is available in 3 repositories. Different repositories provide different sets of data with different licenses. Please go to openpack-dataset @GitHub and follow the instructions.

Sample Data

You can see samples of CSVs and images without downloading them. Sample data (U0209 - Session 5) is available on GitHub (openpack-dataset/data/openpack).

Preprocessed Dataset

Sensors used in the OpenPack dataset have different sampling rates. Therefore, each sensor is stored in a separate file. When you use them, you have to combine them using timestamps assosiated with each record. But we understand that it's not easy for the new users.

Therefore, we prepared pre-processed dataset for the quick trial. IMU data from 4 sensors and work operation labels are combined into one CSV file. This preprocessd dataset is available on zenodo (preprocessed-IMU-with-operation-labels.zip) (Click to Download [515MB]).

For more details, see data-stream/preprocessed.

Full Dataset

See DOWNLOAD.md (@GitHub:openpack-dataset).

Download Commands

Below is a command to download data from zenodo.

# Get a download script.
$ curl -o download_from_zenodo.sh \
  https://raw.githubusercontent.com/open-pack/openpack-dataset/main/release/v1.0.0/download_from_zenodo.sh 
# Create dataset root directory.
$ export DATASET_ROOTDIR_PATH=./data/datasets
$ mkdir -p $DATASET_ROOTDIR_PATH
# Download the dataset.
$ bash download_from_zenodo.sh $DATASET_ROOTDIR_PATH

Terms of Use (Licence)

The OpenPack dataset consists of RGB data (OpenPack Dataset (+RGB)) and other data (OpenPack Dataset). The dataset without RGB data will continue to be provided under "CC BY-NC-SA 4.0". On the other hand, the dataset including RGB data will be provided under a different license for privacy and ethical considerations, as described below. Please check the license of the dataset you use before using it.

OpenPack Dataset

OpenPack Dataset by OpenPack Team is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. This means that you must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use. You may not use the material for commercial purposes.
Creative Commons License

OpenPack Dataset (+RGB)

“OpenPack Dataset (+RGB)” is one of the version that includes RGB data. OpenPack Dataset (+RGB) is licenced under “OpenPack Dataset (+RGB) License” . This means that you may not redistribute this data for any porpose. Also, the usage of this dataset is limited to the academic research only. Use of commercial purpose is prohibited. See the OPENPACK_DATASET_RGB_LICENSE.md (@GitHub:openpack-dataset) for the full license conditions.

Citation

When you use the OpenPack dataset or code please cite the following paper. (Updated on April 21 2024):

Naoya Yoshimura, Jaime Morales, Takuya Maekawa, Takahiro Hara, “OpenPack: A Large-scale Dataset for Recognizing Packaging Works in IoT-enabled Logistic Environments”. Proceedings of IEEE International Conference on Pervasive Computing and Communications (2024). (link)

Tutorials

Notebooks (Colab / Jupyter Notebook)

When you click on the link, a notebook will open in Google Colaboratory. A warning will appear when you run the first cell, but press "continue" to proceed. It is recommended to download the dataset to Google Drive when you run the tutorials on Google Colab. Before starting tutorials, please follow the instructions in the first tutorial (Tutorial - Download OpenPack Dataset to Google Drive) and download the OpenPack dataset from zenodo to your Google Drive.

GitHub

OpenPack Dataset

Dataset Documentations

GitHub

OpenPack Toolkit

Dataset Utilities

GitHub

OpenPack Torch

Sample Code (PyTorch)

YouTube

OpenPack Dataset Channel

Sample Videos and a Tutorial Recording (Japanese)

Paper

Conference

OpenPack: A Large-scale Dataset for Recognizing Packaging Works in IoT-enabled Logistic Environments

N. Yoshimura, J. Morales, T. Maekawa, T. Hara

Proceedings of the IEEE International Conference on Pervasive Computing and Communications (PerCom2024)

Unlike human daily activities, existing publicly available sensor datasets for work activity recognition in industrial domains are limited by difficulties in collecting realistic data as close collaboration with industrial sites is required. This also limits research on and development of methods for industrial applications. To address these challenges and contribute to research on machine recognition of work activities in industrial domains, in this study, we introduce a new large-scale dataset for packaging work recognition called OpenPack. OpenPack contains 53.8 hours of multimodal sensor data, including acceleration data, keypoints, depth images, and readings from IoT-enabled devices (e.g., handheld barcode scanners), collected from 16 distinct subjects with different levels of packaging work experience. We apply state-of-the-art human activity recognition techniques to the dataset and provide future directions of complex work activity recognition studies in the pervasive computing community based on the results. We believe that OpenPack will contribute to the sensor-based action/activity recognition community by providing challenging tasks. The OpenPack dataset is available at https://open-pack.github.io.

Conference

Preliminary investigation of SSL for Complex Work Activity Recognition in Industrial Domain via MoIL

Q. Xia, T. Maekawa, J. Morales, T. Hara, H. Oshima, M. Fukuda, Y. Namioka

Proceedings of the IEEE International Conference on Pervasive Computing and Communications Workshops and other Affiliated Events (PerCom Workshops 2024)

In this study, we investigate a new self-supervised learning (SSL) approach for complex work activity recognition using wearable sensors. Owing to the cost of labeled sensor data collection, SSL methods for human activity recognition (HAR) that effectively use unlabeled data for pretraining have attracted attention. However, applying prior SSL to complex work activities such as packaging works is challenging because the observed data vary considerably depending on situations such as the number of items to pack and the size of the items in the case of packaging works. In this study, we focus on sensor data corresponding to characteristic and necessary actions (sensor data motifs) in a specific activity such as a stretching packing tape action in an assembling a box activity, and try to train a neural network in self-supervised learning so that it identifies occurrences of the characteristic actions, i.e., Motif Identification Learning (MoIL). The feature extractor in the network is used in the downstream task, i.e., work activity recognition, enabling precise activity recognition containing characteristic actions with limited labeled training data. The MoIL approach was evaluated on real-world work activity data and it achieved state-of-the-art performance under limited training labels. (note: Best WIP Award @ PerCom2024)

Conference

Recent Trends in Sensor-based Activity Recognition

T. Maekawa, Q. Xia, R. Otsuka, N. Yoshimura, K. Tanigaki

Proceedings of the IEEE International Conference on Mobile Data Management (MDM)

This seminar introduces recent trends in sensor-based activity recognition technology. Technology to recognize human activities using sensors has been a hot topic in the field of mobile and ubiquitous computing for many years. Recent developments in deep learning and sensor technology have expanded the application of activity recognition to various domains such as industrial and natural science fields. However, because activity recognition in the new domains suffers from various real problems such as the lack of sufficient training data and complexity of target activities, new solutions have been proposed for the practical problems in applying activity recognition to real-world applications in the new domains. In this seminar, we introduce recent topics in activity recognition from the viewpoints of (1) recent trends in state-of-the-art machine learning methods for practical activity recognition, (2) recently focused domains for human activity recognition such as industrial and medical domains and their public datasets, and (3) applications of activity recognition to the natural science field, especially in animal behavior understanding.

Members

Takuya Maekawa
Osaka Univ.

Naoya Yoshimura
Osaka Univ.

Jaime Morales
Osaka Univ.

Special Thanks

K

Kana Yasuda
Annotator

C

Chikako Kawabe
Annotator

M

Makiko Otsuka
Annotator

Y

Yashodmi Kaluarachchi
Annotator

K

Keisuke Tsukamoto
Web Developper

P

Pan Guangyang
Web Developper

K

Kohei Hirata
Web Developper

Y

Yagi
Web Developper

Get Updates from OpenPack Team!

Get the latest information about the OpenPack Challenge or OpenPack Dataset on our mailing list (Google Group), Twitter and GitHub. Please subscribe and follow now!