2021-06-17 19:34:41 +02:00
|
|
|
# merge-psychonautwiki-tripsit-data
|
2021-06-17 21:57:37 +02:00
|
|
|
|
2021-06-17 19:34:41 +02:00
|
|
|
Script to merge data from PsychonautWiki API + pages and TripSit factsheets into one standardized format
|
2021-06-17 20:49:51 +02:00
|
|
|
|
2021-06-25 07:52:47 +02:00
|
|
|
## Installation
|
|
|
|
|
|
|
|
|
|
1. Install virtualenv
|
|
|
|
|
```
|
|
|
|
|
python3 -m pip install --user virtualenv
|
|
|
|
|
```
|
|
|
|
|
2. Create virtual environment
|
|
|
|
|
```
|
|
|
|
|
python3 -m venv env
|
|
|
|
|
```
|
|
|
|
|
3. Activate virtual environment
|
|
|
|
|
```
|
|
|
|
|
source env/bin/activate
|
|
|
|
|
```
|
|
|
|
|
4. Install packages
|
|
|
|
|
```
|
|
|
|
|
python3 -m pip install -r requirements.txt
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
## Usage
|
|
|
|
|
```
|
|
|
|
|
python3 scrape.py
|
|
|
|
|
```
|
|
|
|
|
|
2021-06-17 20:49:51 +02:00
|
|
|
## Output Schema
|
|
|
|
|
|
2021-06-17 21:57:37 +02:00
|
|
|
| Property | Type | Description | Source |
|
|
|
|
|
| ------------------------- | ------------------------ | ------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------- |
|
|
|
|
|
| `url` | `string` | Substance page URL | PsychonautWiki if exists, TripSit factsheet otherwise |
|
2021-06-25 05:36:47 +02:00
|
|
|
| `experiencesUrl` | `null` \| `string` | Erowid Experience Vault page URL | TripSit |
|
2021-06-17 21:57:37 +02:00
|
|
|
| `name` | `string` | Substance Name | shortest display name of PsychonautWiki page header and TripSit API's `pretty_name` |
|
|
|
|
|
| `aliases` | `string[]` | List of aliases (lowercase) | Both |
|
|
|
|
|
| `aliasesStr` | `string` | Comma-separated list of aliases (for fuzzy searching) | Both |
|
|
|
|
|
| `summary` | `string` | Substance summary | TripSit |
|
2021-06-25 05:29:13 +02:00
|
|
|
| `reagents` | `null` \| `string` | Reagent test kit reactions | TripSit |
|
2021-06-17 21:57:37 +02:00
|
|
|
| `classes` | `null` \| `Dictionary` | Substance classifications | PsychonautWiki |
|
|
|
|
|
| `classes.chemical` | `string[]` | Chemical classifications | PsychonautWiki |
|
|
|
|
|
| `classes.psychoactive` | `string[]` | Psychoactive classifications | PsychonautWiki |
|
|
|
|
|
| `toxicity` | `null` \| `string[]` | List of toxicity characteristics | PsychonautWiki |
|
|
|
|
|
| `addictionPotential` | `null` \| `string` | Description of potential for addiction | PsychonautWiki |
|
|
|
|
|
| `tolerance` | `null` \| `Dictionary` | Tolerance information | PsychonautWiki |
|
|
|
|
|
| `tolerance.full` | `null` \| `string` | When full tolerance activates | PsychonautWiki |
|
|
|
|
|
| `tolerance.half` | `null` \| `string` | When tolerance is reduced by half | PsychonautWiki |
|
|
|
|
|
| `tolerance.zero` | `null` \| `string` | When tolerance is back to zero | PsychonautWiki |
|
|
|
|
|
| `crossTolerances` | `null` \| `string[]` | Other substances or classes with cross tolerance | PsychonautWiki |
|
|
|
|
|
| `roas` | `Dictionary[]` | Routes of administration | Both (Note: PsychonautWiki ROAs are prioritized, and TripSit factsheets fill in gaps) |
|
|
|
|
|
| `roas[].name` | `string` | Name of ROA (e.g. 'Oral', 'Smoked') | Both |
|
2021-06-17 22:06:26 +02:00
|
|
|
| `roas[].bioavailability` | `Optional<string>` | Substance's bioavailability for ROA | Both |
|
2021-06-17 21:57:37 +02:00
|
|
|
| `roas[].dosage` | `null` \| `Dictionary[]` | Dosage information for ROA | Both |
|
|
|
|
|
| `roas[].dosage[].name` | `string` | Name of dosage level for ROA (e.g. 'Threshold', 'Strong') | Both |
|
|
|
|
|
| `roas[].dosage[].value` | `string` | Typical dosage for given level of ROA (e.g. '10 - 25 mg', '100 mg +') | Both |
|
|
|
|
|
| `roas[].dosage[].note` | `Optional<string>` | Additional note | PsychonautWiki |
|
|
|
|
|
| `roas[].duration` | `null` \| `Dictionary[]` | Duration information for ROA | Both |
|
|
|
|
|
| `roas[].duration[].name` | `string` | Name of duration item for ROA (e.g. 'Onset', 'Peak') | Both |
|
|
|
|
|
| `roas[].duration[].value` | `string` | Typical duration for given item of ROA (e.g. '30 - 60 minutes', '2 - 4 hours') | Both |
|
|
|
|
|
| `roas[].duration[].note` | `Optional<string>` | Additional note | PsychonautWiki |
|
|
|
|
|
| `interactions` | `null \| Dictionary[]` | Interactions with other substances or classes | TripSit |
|
|
|
|
|
| `interactions[].name` | `string` | Name of other substance or class | TripSit |
|
|
|
|
|
| `interactions[].status` | `string` | Interaction status from TripSit combos | TripSit |
|
|
|
|
|
| `interactions[].note` | `Optional<string>` | Additional note | TripSit |
|
