The time has come to clean up and publish shamotskyi/ua_cbt_stories · Datasets at Hugging Face

From last time:

• 3 batches — unite into 1?
• Columns
  • Weird:
    • READING_LEVEL: is it always gradute student? TODO delete
    • IDs
      • Unnamed: 0 is some kind of ID? That is sequential between batches, batch 2 has 81+ — will delete, it’s a readable ID but oh well
      • annotation_id ??? is it useful? Not sequential in label-studio
      • id another ls id? delete?
    • delete
      • created_at delete
      • lead_time delete
      • price
      • temperature likely wrong
      • updated_at likely wrong
    • comments TODO parse
  • OK
    • annotator: number, safe
    • fixed story
    • generated_story
  • To document
    • BAD_ENDING, NUM_WORDS, N_MAIN_CHARACTERS, N_MINOR_CHARACTERS: all clear
    • story_template used to generate it
    • whynot used to generate it
    • Stories
      • fixed_story
      • generated_story
      • md_story_first_attempt
      • md_story_before_fixing_errors
    • model
    • md_errors_fixed_using_model
    • result: usable/unusable
    • status: if todo then unprocessed
• Fazit no private data but needs prettyin-up

New

• Changes

  • status==todo - dropping all ~10 stories with that
  • others can only be notable for interesting story, renaming
  • Renames
    • 'model': 'model_generation', 'comments': 'annotator_comments'

• Cols documentation:

  • story.id is the same as storyId in UA-CBT and same as md_tale_id_labelstudio!
  • result usable/unusable
  • whynot reasons why not ok
  • model used to generate
  • fixed_story by humansGo

• Stories

  • story_first_attempt, then it was made longer -> md_story_before_fixing_errors
  • then after error fixing it became generated_story,
  • then humans fixed it as fixed_story