用Python比较金标准csv文件和提取值csv文件
用Python比较金标准csv文件和提取值csv文件
提问于 2015-07-02 00:46:04
这是一个数据挖掘任务，在那里，我们将自动地对提取的质量进行评分。有一个金本位csv，它可能由以下字段组成
golden_standard.csv
| id | description | amount | date |
|----|-------------------------|---------|------------|
| 1 | Some description. | $150.54 | 12/12/2012 |
| 2 | Some other description. | $200 | 10/10/2015 |
| 3 | Other description. | $25 | 11/11/2014 |
| 4 | My description | $11.35 | 01/01/2015 |
| 5 | Your description. | $20 | 03/03/2013 |，然后有两个可能的提取结果文件：
extract1.csv
| id | description | date |
|----|-------------------------|------------|
| 1 | Some description. | 12/12/2012 |
| 2 | Some other description. | 10/10/2015 |
| 3 | Other description. | 11/11/2014 |
| 4 | 122333222233332221 | 11/11/2014 |
| 5 | Your description. | 03/03/2013 |extract2.csv
| id | description | amount | date |
|----|-------------------------|---------|------------|
| 1 | Some description. | $150.54 | 12/12/2012 |
| 2 | Some other description. | $200 | 10/10/2015 |
| - | ----------------------- | ----- | ---------- |
| 5 | Your description. | $20 | 03/03/2013 |extract3.csv
| Garbage | More Garbage |
| Garbage | More Garbage | 我想要我的程序报告，即提取1缺少一列，值在第2列中没有正确匹配。
对于第二种情况，我丢失了条目，而且有些行都不匹配。
在最后一种情况下，最终的csv都被搞砸了，但我仍然希望程序能够检测到一些有意义的删减。
有没有人在python中有一些快速而聪明的方法来进行这种比较？
我有我的常规，冗长的逐行和逐行迭代的方式，我可以在这里发布，但我想可能有一个更快，更优雅的毕达通的方式来做到这一点。
任何帮助都是非常感谢的。
回答 1
Stack Overflow用户
回答已采纳
发布于 2015-07-02 03:05:55
pandas免责声明:我的方法使用库.
首先是数据设置。
gold_std.csv
id,description,amount,date
1,Some description.,$150.54,12/12/2012
2,Some other description.,$200,10/10/2015
3,Other description.,$25,11/11/2014
4,My description,$11.35,01/01/2015
5,Your description.,$20,03/03/2013extract1.csv
id,description,date
1,Some description.,12/12/2012
2,Some other description.,10/10/2015
3,Other description.,11/11/2014
4,122333222233332221,11/11/2014
5,Your description.,03/03/2013extract2.csv
id,description,amount,date
1,Some description.,$150.54,12/12/2012
2,Some other description.,$200,10/10/2015
3,Other description.,$25,11/11/2014
5,Your description.,$20,03/03/2013第二，密码。
import pandas as pd
def compare_extract(extract_name, reference='gold_std.csv'):
gold = pd.read_csv(reference)
ext = pd.read_csv(extract_name)
gc = set(gold.columns)
header = ext.columns
extc = set(header)
if gc != extc:
missing = ", ".join(list(gc - extc))
print "Extract has the following missing columns: {}".format(missing)
else:
print "Extract has the same column as standard. Checking for abberant rows..."
gold_list = gold.values.tolist()
ext_list = ext.values.tolist()
# Somewhat non-pandaic approach because possible no same IDs so we're relying
# on set operations instead. A bit hackish, actually.
diff = list(set(map(tuple, gold_list)) - set(map(tuple, ext_list)))
df = pd.DataFrame(diff, columns=header)
print "The following rows are not in the extract: "
print df第三，测试运行。
e1 = 'extract1.csv'
compare_extract(e1)
# Extract has the following missing columns: amount
e2 = 'extract2.csv'
compare_extract(e2)
# Extract has the same column as standard. Checking for abberant rows...
# The following rows are not in the extract:
# id description amount date
# 0 4 My description $11.35 01/01/2015最后，最后一个摘录有点任意性。我想你最好写一个非pandas算法。
页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：
https://stackoverflow.com/questions/31173917
复制相关文章
相似问题
腾讯云开发者
Copyright © 2013 - 2026 Tencent Cloud. All Rights Reserved. 腾讯云 版权所有
深圳市腾讯计算机系统有限公司 ICP备案/许可证号：粤B2-20090059 
粤公网安备44030502008569号
腾讯云计算（北京）有限责任公司 京ICP证150476号 | 京ICP备11018762号