Class: Kiba::Extend::Transforms::Deduplicate::Fields

Inherits:
Object
  • Object
show all
Defined in:
lib/kiba/extend/transforms/deduplicate/fields.rb

Overview

Removes the value(s) of source from targets

Examples:

Multivalue case sensitive deduplication

# Used in pipeline as:
# transform Deduplicate::Fields,
#   source: :x,
#   targets: %i[y z],
#   delim: ";"
xform = Deduplicate::Fields.new(
  source: :x,
  targets: %i[y z],
  delim: ";"
)
input = [
  {x: "a", y: "a", z: "b"},
  {x: "a", y: "a", z: "a"},
  {x: "a", y: "b;a", z: "a;c"},
  {x: "a;b", y: "b;a", z: "a;c"},
  {x: "a", y: "aa", z: "bat"},
  {x: nil, y: "a", z: nil},
  {x: "", y: ";a", z: "b;"},
  {x: "a", y: nil, z: nil},
  {x: "a", y: "A", z: "a"}
]
result = Kiba::StreamingRunner.transform_stream(input, xform)
  .map{ |row| row }
expected = [
  {x: "a", y: nil, z: "b"},
  {x: "a", y: nil, z: nil},
  {x: "a", y: "b", z: "c"},
  {x: "a;b", y: nil, z: "c"},
  {x: "a", y: "aa", z: "bat"},
  {x: nil, y: "a", z: nil},
  {x: "", y: "a", z: "b"},
  {x: "a", y: nil, z: nil},
  {x: "a", y: "A", z: nil}
]
expect(result).to eq(expected)

Single val, case insensitive

# Used in pipeline as:
# transform Deduplicate::Fields,
#   source: :x,
#   targets: %i[y z],
#   casesensitive: false
xform = Deduplicate::Fields.new(
  source: :x,
  targets: %i[y z],
  casesensitive: false
)
input = [
  {x: "a;A", y: "A;a", z: "a;a"},
  {x: "a", y: "A", z: "B"}
]
result = Kiba::StreamingRunner.transform_stream(input, xform)
  .map{ |row| row }
expected = [
  {x: "a;A", y: nil, z: nil},
  {x: "a", y: nil, z: "B"}
]
expect(result).to eq(expected)

Instance Method Summary collapse

Constructor Details

#initialize(source:, targets:, casesensitive: true, delim: nil) ⇒ Fields

Returns a new instance of Fields.

Parameters:

  • source (Symbol) —

    name of field containing value to remove from target fields

  • targets (Array<Symbol>) —

    names of fields to remove source value(s) from

  • casesensitive (Boolean) (defaults to: true) —

    whether matching should be case sensitive

  • delim (nil, String) (defaults to: nil) —

    non-nil is used to split values in source and targets values



76
77
78
79
80
81
82
83
84
85
86
 
# File 'lib/kiba/extend/transforms/deduplicate/fields.rb', line 76

def initialize(source:, targets:, casesensitive: true, delim: nil)
  @source = source
  @casesensitive = casesensitive
  @delim = delim
  getter_params = if @delim
    {fields: targets, delim: @delim}
  else
    {fields: targets}
  end
  @getter = Helpers::FieldValueGetter.new(**getter_params)
end

Instance Method Details

#process(row) ⇒ Object

Parameters:

  • row (Hash{ Symbol => String, nil }) 


89
90
91
92
93
94
95
96
97
98
99
100
101
 
# File 'lib/kiba/extend/transforms/deduplicate/fields.rb', line 89

def process(row)
  sourceval = row[source]
  return row unless sourceval

  targetdata = getter.call(row)
  return row if targetdata.empty?

  sourcevals = split_value(sourceval)
  targetdata.transform_values! { |val| split_value(val) }

  deduplicate(row, sourcevals, targetdata)
  row
end