[HQL] How to analysis Bulk tsv file?

In this case I usually create Hive table with tsv file, and then analysis data with HQL query. First, you need to do is create table CREATE EXTERNAL TABLE 'database_name.table_name' ( id bigint COMMENT 'put column exactly same with tsv file header name', class string, type string, access string, category string ) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde&#3..
Data Engineering/HQL 2020. 5. 14. 16:23
[File] Avro format read with java library 'apache.avro'

Import ‘org.apache.avro:avro:1.9.0’ (choose proper version) follow like this String path = getClass().getResource(path).getPath(); File file = New File(path); DatumReader datumReader = new GenericDatumReader(); DataFileReader dataFileReader = new DataFileReader(file, datumReader); record = dataFileReader.next(record); record.get(key).toString();
Architecture/Back-End 2020. 5. 9. 23:05
[Java library] How to Use FastText4J Library

Save fastest model bin file to AWS S3 Import FastText4J library to Spring java project dependency "com.github.linkfluence:fastText4j:0.2.1" compile('com.github.linkfluence:fastText4j') get fasttext model bin file from S3(Use AWS aws-java-sdk-s3 library) load FastText class with model bin InputStream modelBinary; FastText fastText = FastText.loadModel(modelBinary); call predict function t..
Architecture/Back-End 2020. 5. 8. 21:55
1 2 3 4 ··· 7