Data get and result submit tutorial

Discussion

Home Discussion Board KDD CUP of Fresh Air Data get and result submit tutorial

Patrick Gao

KDD18 Organizer

Data get and result submit tutorial

posted in KDD CUP of Fresh Air

2018-03-02 07:56

### Data API

------------

We provide hourly updated air quality and meteorology data for Beijing and London. .
The data is updated hourly.

For Beijing we provide:
1. Air Quality data
1. Observed Meteorology Data
1. Grid Meteorology Data.

For London we provide :
1. Air Quality data
1.  Grid Meteorology Data

For the difference between observed meteorology data and grid meteorology data, Please visit [here](https://biendata.com/competition/kdd_2018/data/ "here")

##### API detail:
```
https://biendata.com/competition/airquality/{city}/{start_time}/{end_time}/2k0d1d8
```
```
https://biendata.com/competition/meteorology/{city}/{start_time}/{end_time}/2k0d1d8
```
```
https://biendata.com/competition/meteorology/{city}_grid/{start_time}/{end_time}/2k0d1d8
```

##### Example:
For Beijing:
```
Air Quality data:
https://biendata.com/competition/airquality/bj/2018-04-01-0/2018-04-01-23/2k0d1d8
```
```
Observed Meteorology:
https://biendata.com/competition/meteorology/bj/2018-04-01-0/2018-04-01-23/2k0d1d8
```
```
Meteorology Grid Data:
https://biendata.com/competition/meteorology/bj_grid/2018-04-01-0/2018-04-01-23/2k0d1d8
```

For London:
```
Air Quality data:
https://biendata.com/competition/airquality/ld/2018-04-01-0/2018-04-01-23/2k0d1d8
```
```
Meteorology Grid Data:
https://biendata.com/competition/meteorology/ld_grid/2018-04-01-0/2018-04-01-23/2k0d1d8
```

##### API for 48-Hour Weather Forecast Data

Caiyun kindly provides 48 hours weather forecast data for all grid weather points.

```python
http://kdd.caiyunapp.com/competition/forecast/{city}/{start_time}/2k0d1d8

city = ‘bj’ (for Beijing) or ‘ld’ (for London)
time = ‘YEAR-MONTH-DATE’

Example: 
http://kdd.caiyunapp.com/competition/forecast/bj/2018-04-24-00/2k0d1d8
```

##### Here is a piece of code in Python 3 to retrieve data:

```python
import requests

url = 'https://biendata.com/competition/meteorology/bj/2018-04-01-0/2018-04-01-23/2k0d1d8'
respones= requests.get(url)
with open ("bj_meteorology_2018-04-01-0-2018-04-01-23.csv",'w') as f:
    f.write(respones.text)
```

Please Note: 
Please **save the data** after any request and only request for the most recent data

London doesn’t have observed meteorology data

### Submit API

------------

##### The submission link is:
https://biendata.com/competition/kdd_2018_submit/

##### User_id:
Your User_id is being side the logout button, at the top right corner of the website

##### Token:

You can get your token on this page: (after you participate in this competition)

https://biendata.com/competition/kdd_2018/team/my_team/

##### You can download the sample submission file at the [Data](https://biendata.com/competition/kdd_2018/data/  "Data") page:

Here are some top lines of the sample submission file:

| test_id  | PM2.5  | PM10  | O3  |
| :------------: | :------------: | :------------: | :------------: |
|  dongsi_aq#0 | 100  |  75 | 13  |
|dongsi_aq#1|102|87|2|
|dongsi_aq#2|85|85	|2|
|dongsi_aq#3|88|82	|2|
|dongsi_aq#4|95|77|2|
|dongsi_aq#5|95|59|13|
|dongsi_aq#6|95|60|13|
|dongsi_aq#7|103|84|2|
|dongsi_aq#8|109|86|2|

Pattern: {station id} + # + {hour}
station id : The airquality stations needed to predicted.
hour : range from 0 to  47

##### Here is a piece of sample submission code in Python 3:

```python
import requests

files={'files': open('sample_submission.csv','rb')}

data = {
    "user_id": "test_name",
    "team_token": "53dfdfsdfdsfioiuoiuoiu7231asdfa7sdfdfsdf23cdf716e5072",
    "description": 'i am description',
    "filename": "'i am filename",
}

url = 'https://biendata.com/competition/kdd_2018_submit/'
response = requests.post(url, files=files, data=data)

print(response.text)
```

#####After any submission, you will receive submission results, such as:

```python
{“success”: true}  // submission succeeded
{“success”: false}  // submission failed
{“success”: false, “msg”: “user not found: test_name”} // wrong user_id
{“success”: false, “msg”: “Team token incorrect.”} //wrong team token
{“success”: false, “msg”: “Incorrect format.”} // Incorrect format
```

If the submission is successful , you can view your submission on this page:

https://biendata.com/competition/kdd_2018/my-submission/

Leaderboard is here:

https://biendata.com/competition/kdd_2018/leaderboard/

23 comments

Ren

2018-03-31 19:40

Ren reply Ren

2018-03-31 19:45

Reply

0

My above statement is on Beijing’s air quality data. For London, I see the csv file contain history from 2017-01-01:0:00 to 2018-03-09:9:00. The api for london also return data earlist from 2018-03-31-07:00. So there are 22 days of data missing.

Ren

2018-03-31 22:34

望着蓝天的海

2018-04-01 02:18

望着蓝天的海 reply Ren

2018-04-01 02:24

Reply

3

You need to predict the concentration of PM2.5/PM10/O3 for 35 stations in Beijing and PM2.5/PM10 for13 stations in London.

Ren reply Ren

2018-04-01 02:47

Reply

0

Thanks for pointing it out.

G13

2018-04-01 03:07

Biendata Staff

Xinyu reply Ren

2018-04-01 08:54

Reply

8

Thank you very much for this request. We will upload the dataset today or tomorrow. Xinyu from Biendata

Biendata Staff

Xinyu reply G13

2018-04-01 14:57

Reply

0

Please use filename instead of display_name. We have updated the post.

Ren reply Ren

2018-04-01 23:50

Reply

0

Great, thank you!

Yasin Seven

2018-04-02 07:11

Ren reply Ren

2018-04-02 15:10

Reply

2

Hi Xinyu, zhiwuyuan_aq from beijing has no data from either the posted csv file for Feb to March 2018, and the API. Is there anything wrong with data retrieval with this station?

fersebas reply Ren

2018-04-02 17:28

Reply

1

And the observed meteo data? ty

Ren

2018-04-02 20:07

Biendata Staff

Xinyu reply Ren

2018-04-03 09:04

Reply

1

That seems, at least from our side, is not true. We can submit files with decimals. Please send your token, submission file and error message, to <a href="mailto:supprot@biendata.com">supprot@biendata.com</a>

PlrS

2018-04-03 10:56

KDD18 Organizer

KDD18 Organizer

Patrick Gao reply PlrS

2018-04-03 11:18

Reply

0

It ‘ s UTC time

fersebas reply PlrS

2018-04-03 14:01

Reply

0

I see 2945491,TD5,2018-04-03 11:00:00,6.8,,,,, last record now…

Ren reply Ren

2018-04-03 15:41

Reply

0

My email got rejected … Can you take a look at the following error message? Error Icon Message not delivered Your message couldn’t be delivered to <a href="mailto:supprot@biendata.com">supprot@biendata.com</a> because the remote server is misconfigured. See technical details below for more information. The response from the remote server was: 550 5.4.1 [<a href="mailto:supprot@biendata.com">supprot@biendata.com</a>]: Recipient address rejected: Access denied [BL2NAM02FT064.eop-nam02.prod.protection.outlook.com] Thank you!

Ren reply Ren

2018-04-03 16:00

Reply

0

Ah i see, you mis-spelled support… My mail got through now.

2018-04-06 00:12

KDD18 Organizer

KDD18 Organizer

Patrick Gao reply MO

2018-04-07 01:36

Reply

0

Of course yes

Michael Gong

2018-04-07 08:03

MO reply MO

2018-04-07 13:54

Reply

0

Thank you!

HALLEY

2018-04-09 14:04

HALLEY reply G13

2018-04-09 17:07

Reply

0

Hello, i got the same error saying Server Error. I am curious how did you solve that problem? I am using filename instead of Display name but it does not help. Thank you in advance!

jz455

2018-04-10 03:59

WanQuan

2018-04-13 06:32

WanQuan reply WanQuan

2018-04-13 06:46

Reply

0

OK, I know how it works, ignore me.

WanQuan

2018-04-13 07:54

hzhwcmhf

2018-04-13 09:13

crux

2018-04-14 22:37

makingmoney

2018-04-15 12:45

GYJ

2018-04-18 03:30

GYJ reply GYJ

2018-04-18 03:36

Reply

0

Cause I can’t find it in the website <a href="https://biendata.com/competition/kdd_2018/team/my_team/">https://biendata.com/competition/kdd_2018/team/my_team/</a>

rapirent

2018-04-20 19:30

why94nb

2018-04-23 03:56

joy reply Ren

2018-04-24 02:59

Reply

0

can we get the meo data from 2017-01-01 to 2018-03 now?

rapirent reply rapirent

2018-04-25 01:28

Reply

0

solved, the first row should have no space, meaning <code>test_id, PM2.5, PM10, O3</code> will not be accepted

koji reply WanQuan

2018-04-28 16:27

Reply

0

I have a same issue. How do you get data at 0:00 (24:00)?

Vikas Maurya

2018-05-01 05:02

thaihq

2018-05-01 07:51

lbdhr reply thaihq

2018-05-02 23:16

Reply

0

me, too. But I only submitted twice today. Do you solve this problem?

BigPlato reply Ren

2018-05-03 09:19

Reply

0

Sorry to trouble you, do you know how to solve the problem of missing data of zhiwuyuan in Beijing?